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ABSTRACT 


The  type  of  testing  discussed  in  this  report  is  that  which  takes  place  for  every 
field  installation  to  ensure  that  implementation  of  system  components  is 
accomplished  properly  and  that  optimum  system  performance  is  obtained. 
Implementation  testing  is  defined,  it  is  contrasted  with  other  types  of  testing, 
its  motivation  is  discussed,  and  its  planning  requirements  are  outlined.  This 
report  reveals  how  designing  an  implementation  testing  program  in  advance  is 
an  effective  way  of  ensuring  and  accomplishing  the  successful  implementation 
of  the  system  in  the  field,  at  each  location  where  its  components  are  installed. 
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SECTION  I 


INTRODUCTION 


SCOPE 

Once  the  components  of  a  system  have  been  transported  from  the  manu¬ 
facturer's  plant  to  the  emplacement  where  they  will  be  come  part  of  an  opera¬ 
tional  system,  implementation  testing  will  take  place.  This  document  attempts 
to  define  implementation  testing,  contrast  it  with  other  types  of  testing,  discuss 
its  motivation,  and  describe  the  planning  required. 

CONTRAST  WITH  OTHER  TEST  PROGRAMS 

Implementation  testing  takes  place  for  every  field  installation  that  is 
accomplished.  It  is  necessary  to  ensure  that  the  implementation  itself  is 
accomplished  pi’operly  and  that  the  proper  degree  of  system  performance  re¬ 
sults  at  the  end  of  the  implementation.  Implementation  testing  should  not  be 
confused  with  or  act  as  a  substitute  for  design  verification.  The  latter  is  a  part 
of  the  design  and  development  process,  and,  ideally,  it  should  be  completed 
prior  to  any  field  installation.  Similarly,  implementation  testing  should  not  be 
confused  with  Category  I  or  Category  II  testing  which  is  commonly  performed 
on  the  first  aiTiclc  of  an  electronics  system  for  purposes  of  design  and  quality 
acceptance  and  collecting  certain  service  data.  First  article  testing,  also, 
should  be  completed  prior  to  field  installation. 

THE  MOTIVATION  FOR  IMPLEMENTATION  TESTING 

Testing  is  appropriate  upon  implementation  of  any  system  or  element  of 
a  syslcm  whether  or  not  a  copy  of  the  same  system  has  been  tested  previously. 
The  first  article  of  any  system  of  new  components  must  undergo  thorough 
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testing.  The  second  and  subsequent  articles  must  also  undergo  testing  although 

the  expectation  of  failure  should  be  considerably  less,  and,  consequently,  the 

manpower  and  time  programmed  should  not  be  as  great.  The  experience  of  * 

previous  testing  should  yield  simpler  test  methods  and  diagnostic  techniques, 

and  the  emphasis  might  tend  toward  verification  of  proper  performance  rather 

than  discovery  of  sources  of  difficulty.  It  should  be  noted,  however,  that  if  a 

malfunction  does  occur  on  the  second  article,  it  is  just  as  important  to  locate 

and  fix  it  as  in  the  case  of  the  first  article. 

Why  does  a  malfunction  occur  on  a  second  article  if  thorough  testing 
resulted  in  a  well-operating  first  article?  No  matter  how  thoroughly  the  speci¬ 
fications  for  the  system  were  prepared,  no  matter  how  conscientious  the  manu-  ^ 

facturer  was  in  the  performance  of  the  contract,  no  matter  how  well  the  first 
article  performed  during  testing,  the  situation  is  different  with  each  emplacement.  • 

A  technician  in  installing  a  piece  of  equipment  may  have  crossed  wires 
and  connected  them  to  the  wrong  terminals;  atmospheric  conditions  may  cause 
the  radar  background  noise  to  have  different  characteristics;  the  site  location 
and  relative  location  to  other  sites  will  be  different;  and,  the  logical  operation 
of  the  system  at  an  adjacent  site  may  be  different  from  any  used  at  the  first 
article  test  site.  In  addition,  there  are  normally  many  unique  site  adaptation 
parameters  that  must  be  verified  for  accuracy  and  suitability,  and  assurance 
that  insertion  of  the  parameters  did  not  destroy  some  operation  that  was  previ¬ 
ously  satisfactory.  There  are  also,  normally,  several  threshold  parameters 
that  are  not  necessarily  unique  to  any  particular  site,  but  which  must  be  set 
correctly  at  each  site  and  be  verified  for  correct  operation. 

• 

Implementation  testing  will  take  place  whether  or  not  it  is  deliberately 
planned.  One  can  envision,  perhaps,  putting  components  into  place  according  • 

to  site-layout  drawings,  connecting  the  components  according  to  engineering 
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drawings,  and  reading  in  a  computer  program  previously  prepared.  The  day 
must  come,  however,  when  personnel  will  seat  themselves  at  operational  posi- 

•  tions,  turn  on  all  power  switches,  set  in  adaptation  parameters,  and  expect  the 
system  to  operate  according  to  plans  prepared  many  years  previously  and 
modified  continuously  up  to  this  day.  If  no  systematic  testing  has  been  accom¬ 
plished  |)rior  to  this  day,  the  first  system  implementation  test  begins  then. 

THK  NEKD  FOR  TEST  PI^\NNING 

On  delivery  of  a  new  system,  those  concerned  are  interested  in  having 
the  system  operate  at  the  earliest  possible  time.  The  high  expectation  of 
failure  on  the  first  system  trial  suggests  that  the  goal  of  timely  delivery  of  an 

w  operating  system  is  achieved  with  greater  likelihood  and  less  cost  by  planning 

and  conducting  a  systematic  test  program  in  advance,  a  program  which  detects 

•  areas  of  malfunction  with  collection  of  information  that  will  indicate  corrective 
action  or  which  verifies  the  absence  of  malfunction  with  confidence  and  measures 
the  actual  performance  achieved. 

Implementation  testing  should  be  designed  and  conducted  to  answer  the 
following  questions: 

Does  the  system  as  installed  meet  the  specifications? 

Does  the  system,  meeting  the  specifications,  allow  the  job  to  be  done? 

How  well  does  the  system  do  this  job? 

In  designing  a  test  program  one  must,  of  course,  formulate  many  more 
detailed  questions  to  be  answered  for  each  of  the  three  general  questions.  With 

•  proper  design,  one  can  often  obtain  answers  to  all  three  major  questions  with 
the  same  test  plan.  In  testing  for  the  first  question,  however,  one  needs  to  be 

•  sure  that,  if  the  answ^er  is  in  the  negative,  he  has  obtained  sufficient  informa¬ 
tion  to  til  low  determination  and  correction  of  the  deficiency.  The  answers  to 
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the  third  question  are  usually  obtained  as  byproducts  to  testing  specifically 
designed  to  cover  the  first  two  questions,  and  can  be  augmented  by  collection 
of  information  throughout  the  normal  course  of  operations  following  delivery. 

Proper  test  design  requires  comprehensive  and  realistic  planning:  among 
the  major  areas  that  should  be  treated  are  the  total  time,  the  manpower,  the 
instrumentation,  and  the  external  resources  required.  It  generally  happens 
that  a  computer-based  air  defense  system  is  installed  at  an  existing  manual 
air  defense  site.  It  is  expected  that  the  radar  and  other  operational  resources 
of  that  site  will  be  used.  Test  planning  must  recognize  the  need  of  the  opera¬ 
tional  elements  to  continue  operating.  Too  often  a  test  plan  for  installation  of 
a  computer-based  system  will  assume  that  the  system  manufacturer  has  use 
and  control  of,  say,  the  radar  24  hours  a  day,  7  days  a  week.  In  an  operational 
environment,  he  will  be  fortunate  to  get  use  of  the  radar  for  one  hour  a  day. 

Test  planning  should  take  this  into  account. 

Failure  to  consider  all  the  pertinent  factors  in  test  design  usually  results 
in  nothing  more  serious  than  delays  in  delivery.  Delays  in  delivery,  however, 
can  have  serious  impact.  Contractor  manpower  costs  continue  in  significant 
amounts  without  useful  results,  and  operational  commands  support  manning 
for  two  systems  over  the  delay  period  at  significant  cost.  False  impressions 
based  on  capability  at  expected  delivery  date  can  generate  undue  lack  of  confi¬ 
dence  in  the  system  performance  of  the  system  untimately  delivered. 
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SECTION  II 


CONSTRUCTION  OF  THE  TEST  PLAN 


Let  us  now  try  to  construct  an  implementation  test  plan  for  a  computer- 
based  air  defense  system.  This  should  be  accomplished  in  accordance  with  the 
general  objectives  of  the  implementation  test  program,  (see  p.  1).  Specific 
questions  must  be  constructed  for  each  of  the  general  objectives. 

TEST  OBJECTIVES 

Tests  for  Compliance  with  the  Specifications 

For  the  first  general  objectives,  of  course,  one  needs  to  have  the  specifi¬ 
cation  before  the  specific  questions  can  be  completed.  Some  examples  can  be 
given,  however,  to  illustrate  the  type  of  operations  that  should  be  checked. 

If  a  Link-1  message,  say  S-4,  is  required  to  be  sent  between  a  pair  of 
adjacent  sites,  the  following  series  of  steps  is  indicated. 

(a)  Insure  that  the  telephone  lines  are  installed  between  the  two  sites 
concerned,  and  that  they  have  the  specified  characteristics.  Of 
particular  concern  is  the  requirement  that  the  specified  number 
of  bits  per  second  can  be  passed  from  one  computer  to  the  other 
with  only  the  allowable  loss  of  bits. 

(b)  Insure  that  the  format  generated  within  the  sending  computer  is 
received  in  the  same  format  in  the  other  computer.  One  might 
make  this  check,  for  example,  by  including,  in  a  sample  message 
format,  zeros  in  all  information  items  except  one.  That  information 
item,  perhaps  bits  9  to  12,  might  be  filled  with  ones.  A  check  of  the 
receiving  computer  could  then  be  made  to  see  that  all  ones  were 
received  for  the  information  item  of  bits  9  to  12,  and  that  zeros 
occurred  elsewhere.  A  complete  check  of  this  step  might  require 
that  all  possible  combinations  of  zeros  and  ones  were  examined. 

It  should  suffice,  in  most  cases,  to  check  only  a  small  sample  of 
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the  possibilities.  The  sample  should  include  examination  of  the 
information  items  which  might  be  most  important  in  operations. 

Testing  for  the  second  and  third  objectives  and,  indeed,  monitoring 

of  performance  during  operations,  should  reveal  discrepancies  of 

significance,  if  any,  not  uncovered  by  this  specific  test.  * 

(c)  Insure  that  the  message  is  generated  according  to  prescribed  rules 
for  area,  timing,  or  other  criteria.  To  accomplish  this  step,  one 
might  attempt  to  crosstell  several  tracks  with  the  S-4  message  by 
placing  these  tracks  for  test  purposes  at  various  points  within  and 
without  the  boundaries  of  the  areas  for  automatic  crosstell.  At  the 
receiving  site,  a  check  can  be  made  to  insure  that  no  tracks  outside 
of  the  area  are  received  and  that  tracks  meeting  the  other  criteria 
are  received  within  the  prescribed  area  at  the  rate  of  message 
generation,  say  one  per  scan,  specified. 

In  examining  the  generation  resulting  from  other  criteria,  for 
example,  identity  or  request  for  a  message  from  another  site, 

steps  should  be  taken  to  insure  that  an  identification  changing  the  ’ 

eligibility  of  a  track  for  crosstell  results  in  the  proper  response, 
and  that  a  request  for  a  message  results  in  the  proper  message 
being  sent. 

(d)  Insure  that  the  information  content  is  set  properly  according  to  rules 
and  definitions.  In  some  cases,  a  very  simple  test  is  involved.  For 
example,  with  identification  content,  one  can  merely  assign  success¬ 
ively  all  the  possible  identification  classifications,  and  check  to 

see  that  the  correct  identification  classification  is  included  in  each 
message  being  crosstold.  It  is  slightly  more  complex  to  insure 
the  proper  setting  of  an  information  item  such  as  data  quality.  In 
this  case,  one  must  examine  the  history  of  tracking,  looking  at  the 
external  factors  that  determine  data  quality.  If  the  data  conditions 
are,  in  fact,  poor,  the  data  quality  transmitted  should  be  poor;  and, 
in  a  similar  fashion,  if  the  data  quality  transmitted  is  good,  then 
the  data  history  on  the  track  concerned  should,  in  fact,  be  good. 

(e)  Insure  that  the  information  is  interpreted  and  used  properly  at  the 

receiving  site.  The  best  example  here  is,  again,  that  of  data 
quality.  At  the  receiving  site  there  are,  presumably,  some  condi¬ 
tions  under  which  crosstold  information  of  good  quality  tracks  • 

should  be  incorporated,  other  conditions  under  which  crosstold 

data  of  poor  quality  tracks  should  be  incorporated,  and,  perhaps, 

some  conditions  under  which  only  part  of  the  information  received  ♦ 
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.  should  be  used.  The  check  then  must  insure  that  the  information 

is  properly  decoded  and  that  it  is  used  in  accordance  with  the  rules 
for  the  conditions  existing  at  the  receiving  site  at  the  time  of 

#  receipt.  Thus,  the  examination  must  be  concerned  not  only  with 

the  message  itself,  but  with  the  conditions  at  the  receiving  site. 

Testing  of  the  sort  indicated  in  this  section  can  easily  get  out  of 
hand  if  one  tries  to  insure  correct  operation  under  every'  possible 
situation.  It  is  important  for  efficiency,  confidence,  and  saving 
of  time  in  the  total  imjdementation  process  to  perform  this  kind 
of  testing  for  at  least  a  few  of  the  commonly  expected  situations 
so  that  obvious  malfunctions  installation  errors,  or  data  processing 
errors  can  be  eliminated  before  proceeding  to  a  higher  level  of 
testing.  A  continuous  search  for  faults  of  this  nature  should  be  made, 
as  a  byproduct,  for  other  situations  encountered  during  high-level 
test  programs. 

Another  illustration  of  a  check  to  see  that  the  sj^stem  as  installed 
meets  the  specifications  might  be  the  following:  if  the  tracking 
logic  includes,  say  bifurcatious  tracking,  one  should  check  to  see 

•  that  a  secondary  track  is  formed  if  and  only  if  the  prescribed 

history  of  data  correlation  is  achieved;  that  a  secondary  track 
becomes  a  primary  track  if  and  only  if  the  proper  history  of  data 
correlation  is  achieved;  and  finally,  that  the  excess  track  is  dropped 
if  and  only  if  the  proper  history  of  data  correlation  is  achieved.  A 
check  of  this  nature  requires  detailed  examination,  in  a  few  represen¬ 
tative  situations,  of  the  data  correlation  and  the  process  of  the 
tracking  equations.  An  examination  of  this  sort  at  each  site  being 
implemented,  using  live  radar  information,  can  be  accomplished  in 

a  fairly  simple  manner,  and  may  save  much  time  and  effort  in 
subsequent  tests  to  evaluate  tracking  capability,  say,  on  maneuver¬ 
ing  aircraft. 

Although  the  specifications  for  the  system  may  not  always  prescribe 
tolerances  on  the  registration  of  target  information  between  two 
adjacent  radar  sites,  they  probably  should.  (Registration  is  defined 
here  to  mean  the  degree  to  which  two  separate  sites  can  report  a 
single  target  in  such  a  fashion  that,  after  coordinate  conversion, 
the  location  reported  matches  that  computed  at  the  other  site.  ) 
Whether  the  specifications  indicate  a  tolerance  for  the  discrepancy 
in  position  reported  or  not,  tests  should  be  conducted  to  measure 

*  the  separation  and  to  uncover  the  errors  responsible  for  significant 

separation,  if  any.  The  types  of  errors  commonly  responsible  for 
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excessive  separation  are  errors  in  (1)  coordinate  conversion  data 

processing,  (2)  the  coordinates  of  site  location  in  one  computer  or 

the  other  or  both;  (3)  common  alignment  of  radars  to  true  north; 

or,  finally,  (4)  in  the  range  timing  of  one  or  another  radar.  In  • 

addition,  faulty  height  information  on  test  targets,  if  contained  in 

one  computer  or  the  other,  may  cause  apparent  registration  errors 

and,  hence,  should  be  guarded  against. 

Although  the  tests  that  are  normally  conducted  to  measure  registra¬ 
tion  errors  require  a  fair  amount  of  test  control,  data  reduction 
and  analysis,  the  results  of  this  analysis  ordinarily  indicate,  in 
a  straightforward  manner,  the  correction  necessary  to  improve 
the  registration.  In  addition,  incorporating  at  least  two  steps  into 
the  test  process  can  increase  the  confidence  in  achieving  success- 
fult  registration  at  an  early  time.  The  two  principal  steps  would 
be  a  static  test  wherein  stationary  targets  are  passed  from  one 
site  to  another  and  back  again  to  detect  obvious  errors  in  the 
implementation  of  the  coordinate  conversion  function,  and  a  series 
of  tests  on  a  live  target  aircraft  to  measure  the  actual  separation. 

Finally  some  of  the  most  significant  but  trivial  tests  that  can  be  • 

accomplished  to  meet  the  first  general  objectives  are  mentioned. 

These  consist  of  pushing  each  button  at  a  console  and  generating 
each  display  at  each  console  to  see  if  the  buttons  cause  the  specified 
computer  responses  and  if  the  displays  contain  the  proper  informa¬ 
tion.  In  many  test  programs,  this  is  the  extent  to  which  planned 
checkout  takes  place.  Invariably,  tests  of  a  more  sophisticated 
nature  of  the  sort  indicated  above  subsequently  manifest  themselves 
as  being  necessary.  They  can  then  be  accomplished  only  with  great 
difficulty,  and  waste  results  because  of  lack  of  planning  and  conduct 
in  an  orderly  fashion. 

All  the  checks  that  should  be  made  to  verify  that  the  system  is 
properly  installed  have  not  been  covered.  A  review  of  the  specifica¬ 
tions  for  any  particular  system  should  indicate  the  important 
checks  to  be  made  before  going  on  to  further  testing.  In  addition, 
it  is  always  advisable  to  continue  looking  for  deviations  from  speci¬ 
fications  as  subsequent  testing  takes  place.  This  can  be  done  in  a 
planned  fashion  so  that  once  the  more  important  checks  have  been 
made,  the  remainder  can  be  accomplished  simultaneously  with 
performance  testing  without  any  expectation  of  seriously  affecting 
subsequent  test  conduct.  In  fact,  if  performed  simultaneously  * 
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with  sul)SO(|uent  testing,  one  might  find  th;it  the  system  operating 
with  the  error  still  allows  the  job  to  be  done  and,  heiiee,  there  is 
no  need  to  correct  the  defieieney. 

Tests  to  X'erifv  Performance 

The  s(*cond  general  objective  is  to  determine  whether  or  not  the  system 
with  its  deficiencies  removed  still  allows  the  mission  to  be  accomplished.  To 
aiiswei  Ihis  (juestion,  of  course,  one  needs  to  think  al)out  the  job  the  system  i^ 
hying  to  accomplish.  For  an  air  defense  system,  the  principal  task  is  to  deiu' 
ac’cc'ss  to  enemy  airc*raft  over  friendlj'  territory.  Ultimately,  the  objective 
is  to  d(  stroy  the  enemy  aircraft  before  they  ean  laujieh  weajx)ns  that  might 
damage'  or  destroy  elements  of  the  environment  that  are  being  protected.  Current 
air  (lebmse  systems  have  two  principal  means  of  destroying  enemy  aircraft: 
the  surface-to-air  missile,  and  the  manned  interceptor. 

first  in  the  order  of  discussion  is  the  surface-to-air  missile  (SAM).  The 
guidaiK  c'  and  control  of  a  SAM  is  not  normallj^  the  res})onsibility  of  the  ground 
envi  ronment  of  c'oncc'rn.  The  SAM  control  units  must  be  provided  with  suffieicait 
target  information  so  that  a  SAM  can  be  guided  to  the  target  at  close  to  its 
ma.ximum  range  in  the  direction  from  which  the  enemy  penetration  is  taking  place. 
'The  nu'asure  of  success,  therefore,  should  be  the  timeliness  with  which  informa¬ 
tion  of  target  position  can  be  transmitted  with  sufficient  accuracy  so  that  the 
missile  target  tracking  radar  ean  loek-on  to  the  target  aircraft.  The  question 
which  must  be  answered  then  is  this:  can  the  ground  environment  assign  targets 
to  the  SAM  units  in  sufficient  time  and  with  sufficient  accuracy  so  that  the 
missile  (‘an  destroy  the  hostile  aircraft  with  its  nominal  probability  of  kill? 

The  sjiccMfic  test  measurements  should  be  made  against  criteria  determined  by 
the  opera! ionii  1  emuronment  and  by  the  desired  probability  of  kill. 
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With  respect  to  the  manned  interceptor,  the  task  of  the  ground  environ¬ 
ment  is  to  place  the  interceptor  in  such  a  position  that  its  own  weapons  system 
can  destroy  the  target  with  a  high  probability  of  kill.  The  measure  of  success 
of  the  ground  environment  then  is  the  ability  to  place  the  interceptor  on  a  head¬ 
ing  so  that  it  can  have  optimum  or  near  optimum  crossing  angle,  displacement 
from  the  target,  and  altitude  separation  for  attaining  the  objective  of  completing 
the  intercept  as  far  fonvard  in  the  direction  of  enemy  penetration  as  feasible. 
Thus,  the  ground  environment  must  be  able  to  get  the  interceptor  off  the  ground 
quickly  to  give  it  a  course  toward  the  target  so  that  fuel  and  time  will  not  be 
wasted  in  gaining  a  position  on  an  attack  heading  such  that  the  interepet  can  be 
accomplished.  Normally,  the  desired  tactics  and  profiles  and  conditions  under 
which  each  of  these  may  be  used  are  specified.  For  each  tactic  and,  when 
designated,  each  profile,  the  task  is  to  insure  that  the  interceptor  is  placed 
on  a  specified  heading  within  a  specified  tolerance,  at  a  specified  altitude 
within  specified  tolerance,  at  a  certain  distance  from  the  target  within  a  speci¬ 
fied  tolerance,  with  a  reasonable  time  or  distance  of  target  penetration,  again, 
within  a  specified  tolerance. 

Assuming  that  the  system  can  satisfactorily  designate  targets  to  the  SAM 
units  and  that  they  can  satisfactorily  guide  interceptors  to  a  position  where  they 
can  kill  targets,  the  remaining  measure  of  success  of  the  ground  environment 
is  whether  or  not  it  can  determine  the  presence  and  location  of  all  hostile  tar¬ 
gets  and  choose  among  these  the  ones  which  should  be  committed  to  each  avail¬ 
able  weapon.  Ideally,  the  ability  of  the  target  to  penetrate  an  environment 
completely  is  accomplished  only  by  deficiencies  in  our  weapons  inventory  or 
deployment  or  by  an  allowable  deviation  from  a  100  per  cent  kill  probability. 

To  insure  that  a  system  is  good,  there  are  several  tests  that  should  be 
passed  successfully  as  necessary  steps  toward  accomplishing  ultimate  success. 


The  first  of  these  is  detection  of  the  target  aireraft.  The  system  should  obviously 
be  able  to  deteet  the  presenee  of  all  aireraft  that  are  within  the  radar  eoverage 
speeifiecl  for  the  environment  during  the  entire  time  that  the  aireraft  are  in  this 
eoverage.  The  system  should  be  able  to  traek  eaeh  of  these  targets  with  the 
same  traek  and  traek  number  in  a  eontinuous  fashion  throughout  the  time  that 
they  are  in  the  radar  eoverage  of  the  total  environment.  The  system  should 
allow  identifieation  of  eaeh  of  these  targets  and  maintain  this  identifieation  so 
long  as  it  is  valid  and  so  long  as  the  target  is  being  traeked.  The  system  should 
allow  determination  of  altitude  of  the  target  with  suffieient  aeeuraey  so  that  the 
intereepts  ean  be  aeeomplished  sueeessfully .  The  system  should  allow  deter¬ 
mination  to  be  made  in  a  timely  fashion  of  what  targets  should  be  taken  under 
fire  and  what  weapons  should  be  used  against  targets  so  indieated.  Eaeh  of 
these  tests  should  be  measured  against  eriteria  whieh  provide  for  reasonable 
toleranee  but  still  allow  the  desired  system  kill  probability  to  be  aehieved. 

Ideally,  one  might  produee  a  mathematieal  model  with  various  threat  situa¬ 
tions  that  would  allow  statistieal  determination  of  the  toleranee  to  be  plaeed  on 
eaeh  of  these  steps  so  that  the  desired  system  capability  can  be  achieved, 
assuming  each  of  these  steps  meet  the  assigned  criteria.  In  practice,  a 
criteria  for  each  of  these  steps  is  determined  arbitrarily  based  on  judgement 
of  what  seems  reasonable  for  the  equipments  involved,  the  environmental 
characteristics,  and  needs  of  operational  personnel.  VVHiether  or  not  the 
criteria  are  determined  arbitrarily,  they  should  still  be  established,  and  tests 
should  be  conducted  to  see  that  the  system  as  installed  meets  these  criteria. 

When  it  does,  and  only  when  it  does,  can  it  be  said  that  the  system  has  been 
installed  successfully. 
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Consider,  for  example,  one  specific  test  that  should  be  conducted  for 
determining  success  of  target  maneuvers  in  a  system  with  bifurcations  track¬ 
ing.  For  implementation  testing,  several  target  aircraft  should  be  flown  in 
straight-line  courses,  followed  by  maneuvers  of  certain  magnitude  at  a  certain 
rate.  The  criteria  designed  arbitrarily  might  be  that  the  target  should  be 
tracked  successfully  throughout  a  certain  percentage  of  each  type  of  maneuver, 
that  it  should  stabilize  on  a  new  straight-line  heading  within  a  certain  toler¬ 
ance  within  a  certain  time,  and  that  during  each  straight  leg  the  track  should 
remain  with  the  aircraft  in  the  sense  that  the  velocity  is  that  of  the  aircraft 
within  a  specified  tolerance,  and  the  position  is  that  of  the  aircraft  radar  data 
within  a  specified  tolerance  for  a  specified  percentage  of  time  and  with  a  speci¬ 
fied  maximum  amount  of  operator  correction  actions.  During  a  test  for  system 
capability,  it  must  be  remembered  that  operators  are  part  of  the  system  and  that 
they  are  provided  with  actions  to  correct  situations  with  which  the  automatic 
data  processing  equipment  cannot  contend.  If  this  test  can  be  passed  satisfactor¬ 
ily  and  repeatedly,  and  if  the  tolerances  are  satisfactory  to  achieve  the  desired 
system  capability,  then  it  can  be  said  that  with  respect  to  this  particular  step 
the  system  as  installed  and  meeting  specifications  as  prescribed  does  the  job 
for  which  it  was  intended.  K  it  does  not  pass  the  test,  then  either  there  is 
still  a  hidden  error  not  uncovered  during  previous  testing,  or  the  specifications 
were  not  adequate  to  allow  the  job  to  be  done.  Both  of  these  situations  commonly 
occur.  However,  by  planning  and  conducting  a  competent  test  program  with 
the  first  objectives,  and  by  preparing  the  system  specifications  in  a  thorough 
and  detailed  fashion,  the  chance  of  success  in  passing  a  test  of  this  kind  is 
considerably  enhanced  with  a  corresponding  increase  in  confidence  in  the  sys¬ 
tem  delivered  and  saving  in  cost  and  time  in  the  total  implementation  effort. 


V 

♦ 
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Tests  to  [evaluate  Performance 


The  third  general  objective  is  to  determine  how  well  the  system  does  the 
job  for  which  it  was  bought.  In  our  previous  testing,  the  principal  objective  was 
to  assure  that  system  performance  met  desired  criteria.  Assume,  as  a  desired 
criteria  for  tracking  continuity  that  the  track  should  be  within  5  per  cent  of  air¬ 
craft  position  90  per  cent  of  the  time  that  the  aircraft  is  within  radar  coverage 
of  the  environment,  while  maintaining  the  same  track  number  and  identification 
throughout.  The  measurements  may  actually  indicate  that  for  95  per  eent  of 
the  time  the  track  heading  is  within  2  per  cent,  the  track  speed  within  10  knots, 
and  the  ti'ack  position  within  1-1/2  miles  of  the  corresponding  parameters  of 
the  aircraft.  This  information  is  useful.  It  ean  be  obtained  as  a  direct 
byi)roduct  of  testing  for  the  second  general  objective,  and  ean  be  collected 
V  rci)eatedly  after  system  operations  begin.  It  ean  be  useful  to  operational 

l)crsonnel  to  understand  better  the  system  capabilities,  and  it  ean  be  useful 
to  detect  subsequent  degradation  of  system  performance.  For,  in  spite  of  the 
specifications  or  the  previously  determined  criteria  of  success,  deviation  from 
higher  performance  sustained  over  some  period  of  time  indicates  that  some 
unnecessary  malfunction  has  been  introduced  into  the  system.  This  malfunction 
can  then  be  located  and  corrected  to  return  the  system  to  its  proven  capability 
of  higher  level  of  performance. 

TFST  FACILITIES 

hi  addition  to  specifying  each  of  objective  for  the  test  program  and  the 
method  of  analysis  for  each  objective,  the  test  plan  should  treat  other  import¬ 
ant  topics:  the  internal  facilities  of  the  ground  environment  to  be  used  for 
each  test,  and  the  external  facilities  required  for  each  test  indicating  the 
y  COO rd i na tion  neces sa ry . 
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Internal  Facilities 


Internal  facilities  break  down  into  four  categories.  The  first  should  be 
an  identification  of  the  normal  operating  facilities,  for  example;  consoles, 
computer  programs,  and  sensor  devices  that  need  to  be  utilized  in  each  test. 

The  second  category  consists  of  recording  equipment  and  other  test 
instrumentation  necessary  to  collect  information  that  is  not  necessarily  a  part 
of  normal  operations,  but  which  is  essential  for  the  test  program.  This  may, 
in  fact,  include  additional  display  equipment  and  radio  equipment  for  control 
of  test  aircraft  and  safety  monitoring.  It  will  also  include  recording  devices 
indicating  specific  information  to  be  recorded  so  that  the  necessary  data  for 
test  analysis  is  available.  In  a  computer-based  system,  recording  devices 
planned  should  normally  include  not  only  photographs  and  manual  logs,  but 
also  digital  recording  of  information  contained  in  the  computer  stores  so  that 
specific  data  processing  situations  can  be  reconstructed.  Some  of  the  exam¬ 
ples  given  before  suggest  recording  of  Link  1  messages  as  transmitted  and  as 
received,  recording  of  specific  positions  of  data  correlating  with  specific 
tracks,  recording  of  track  positions  and  velocity,  and  recording  of  operator 
actions,  to  suggest  a  few.  Operator  actions  can,  perhaps,  be  recorded  by 
manual  logs;  the  other  information  is  almost  unavailable  except  through  record¬ 
ing  of  digital  data  in  computer  storage.  In  any  event,  the  specific  information 
to  be  obtained  by  the  various  means  of  instrumentation  should  be  determined  by 
examination  in  terms  of  the  intended  method  of  analysis  of  each  test  objective, 
and  by  assuring  that  the  information  necessary  for  the  analysis  is  obtained  by 
some  means  and  that  the  necessary  test  equipment  is  available  for  the  test 
program. 

The  third  category  consists  of  normal  operator  personnel  required  to 
participate  in  the  program  for  each  test. 
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The  fourth  category  consists  of  normal  operator  personnel  required  for 
test  conduet  and  safety  operations  in  addition  to  the  operating  personnel. 

External  Facilities 

The  external  facilities  which  should  be  indicated  in  the  test  program 
include  the  test  aircraft,  use  of  manual  facilities  as  necessary  for  monitoring 
purposes,  coordination  with  air  traffic  control  facilties  for  test  purposes, 
participation  of  SAM  units  for  test  purposes,  and  others.  The  extent  to  which 
these  arc  required  during  the  test  program  should  also  be  indicated  by  review 
of  a  method  of  analysis  for  each  objective.  Again,  it  is  important  that  the 
requirements  in  this  area  be  understood  and  established  well  in  advance  so  that 
the  people  who  must  provide  the  support  have  an  opportunity  to  plan  for  it 
adequately. 

TEST  RESULTS 

The  application  of  test  results  is  an  important  part  of  planning  an  efficient 
test  program.  First  of  all,  it  may  be  determined  that  the  results  of  some 
particular  test  are  not  applicable.  If  possible,  this  should  be  determined  before 
the  test  is  conducted.  Second,  preplanning  the  application  of  the  results  should 
decrease  the  elapsed  time  between  the  conduct  of  a  test  and  the  correction  of  a 
deficiency,  if  any.  For  example,  with  a  test  of  radar  registration,  a  test  may 
be  planned  well,  conducted,  the  errors  measured,  and  the  existence  of  a 
significant  separation,  due  to  a  faulty  site  location  parameter,  discovered. 

Thus  far  the  program  has  proceeded  in  accordance  with  the  plan.  Obviously, 
the  next  step  is  to  determine  the  correct  parameter  and  insert  it  into  the 
appropriate  computers.  There  should  be  a  plan  for  accomplishing  this  as  a 
part  of  the  test  program  so  that  as  soon  as  the  error  is  determined,  the  correc¬ 
tion  can  be  implemented.  Similarly,  if  a  large  amount  of  information  is 
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obtained  on  the  performance  of  the  system  in  various  functions,  the  informa¬ 
tion  should  be  provided  in  a  consolidated  and  useful  fashion  to  the  operator 
personnel  who  might  benefit  from  understanding  the  system  capabilities.  The 
information  will  not  be  of  much  use  if  it  is  retained,  say,  only  in  a  contractor's 
files.  A  plan  for  compilation  and  distribution  of  the  test  information  in  a  useful 
way  well  in  advance  is  an  important  part  of  a  test  program. 

PHASING  PLAN 

Finally,  the  test  plan  should  include  relative  schedule  or  phasing  informa¬ 
tion  which  indicates  the  scope  of  effort  and  extent  of  resources  required 
throughout  the  test  program.  This  should  be  done  even  if  at  the  time  of  prepara¬ 
tion  the  starting  date  of  implementation  is  not  known.  The  schedule  should  be 
based  on  an  examination  of  each  of  the  test  objectives,  on  the  amount  of 
personnel  and  other  resources  available  in  the  test  program,  on  a  certain 
expectation  of  failure  (particularly  in  the  early  phases),  a  certain  elapsed 
time  for  correction  of  deficiencies  uncovered,  a  reasonable  expectation  of 
availability  of  operational  facilities,  and  a  reasonable  rate  of  aircraft  support. 
The  last  items  must  consider  other  aircraft  obligations,  aircraft  abort  rates, 
and  the  likelihood  of  simultaneous  availability  of  both  the  aircraft  and  the 
ground  environment  under  test. 

Determination  of  the  phasing  and  the  resources  required  is  likely  to 
reveal  that  the  total  test  effort  is  quite  large.  Adequate  personnel  and  other 
resources  to  accomplish  the  effort  will  likely  be  available  only  if  the  require¬ 
ments  are  known  well  in  advance  of  the  need.  A  well-coordinated,  well  thought 
out  realistic  schedule  can  have  major  benefits  in  the  efficient  use  of  personnel 
and  other  resources  and  can,  consequently,  save  time  and  money  and  increase 
total  confidence  in  the  system. 


SECTION  III 


CONCLUDING  REMARKS 

Ideally,  test  planning  is  a  part  of  the  total  system  design.  The  system 
design  should  be  directed  to  yield  the  desired  performance.  The  latter  should 
be  specified  in  such  a  way  that  one  can  determine  whether  or  not  it  has  been 
achieved.  Thus,  the  specific  test  objectives  are  basically  constructed.  The 
test  instrumentation  is  often  an  integral  part  of  the  operational  system  equip¬ 
ment  and  program.  It  needs  to  be  treated  as  part  of  the  total  S3’Stem  design 
considering  access  to  essential  data,  the  large  overlap  with  system  training 
requirements,  and  the  needs  of  operational  exercises. 

This  program  has  been  discussed  in  terms  of  implementation  testing  mostly 
from  a  testing  point  of  view.  The  intent  is  not  to  suggest  that  testing  should  be 
done  just  for  its  own  sake.  In  fact,  unless  it  can  be  demonstrated  how  results 
of  a  test  are  going  to  be  useful  to  the  system,  the  test  should  not  be  conducted. 

In  a  sense,  the  testing  described  is,  in  fact,  the  implementation  of  the  system. 
Designing  a  program  of  this  sort  in  advance  is  a  way  of  ensuring  and  accomplish¬ 
ing  the  successful  implementation  of  the  system  in  the  field,  at  each  location 
where  its  components  are  installed. 
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